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Abstract. The replica method is applied to a neural network model with state- 
dependent synapses built from those patterns having a correlation with the state of 
the system greater than a certain threshold. Replica-symmetric and first-step replica- 
symmetry-breaking results are presented for the storage capacity at zero temperature 
as a function of this threshold value. A comparison is made with existing results based 
upon mean-field equations obtained by using a statistical method. 



INTRODUCTION 



It is standard knowledge by now that for the Hopfield model |l|] the Hebb rule 
leads to a critical storage capacity a c = 0.138 @ while for these type of models with 
quadratic interaction the optimal storage capacity is a c = 2 ||. This is due to the fact 
that the contribution of the noise caused by the weakly correlated patterns becomes 
larger than the signal of the condensed patterns as a increases. In order to lift this 
limitation of the Hebb rule a model with state-dependent synapses has been discussed 
recently [f|, The idea thereby is to introduce a threshold r\ cutting out of the Hebb 
rule all patterns whose correlations with the state of the system are smaller than this 
threshold |J. These authors propose an energy function for this state-dependent synapse 
(SDS) model and derive the corresponding fixed-point equations using the so-called 
heuristically motivated statistical mean-field scheme developed in |6|, [7| (see also ||). In 
the case of the Hopfield model (r/ = 0) this statistical derivation leads to the same results 
as those derived using a replica symmetric mean- field theory approach || . Solving these 
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fixed-point equations one finds, for example at zero temperature, an increase in the 
storage capacity from a c = 0.138 for 77 = up to, e.g., a c = 0.17 for 77 = 1. A similar 
effect has been found for the recognition of temporal sequences H and for non-monotonic 
Hopfield models |T0| . 

In this paper we apply the replica method to the zero temperature capacity problem 
of the SDS model. The aim thereby is twofold. First we want to find out whether the 
replica symmetric (RS) fixed-point equations derived by the standard replica approach 
again coincide with the fixed-point equations found with the statistical method. Second 
since we expect that the RS results are unstable at zero temperature, we want to 
determine the effects of a first-step replica-symmetry breaking (RSB1) on the capacity. 

Somewhat surprisingly, we find that the RS fixed-point equations are different from 
the results obtained in ||. This is due to the fact that for 77 7^ the assumptions 
made in || that both the overlap with the non-condensed patterns as well as the noise 
induced by these non-condensed patterns have a Gaussian distribution are incompatible. 
Keeping only the (standard) assumption that the noise is Gaussian we can improve the 
calculations using the statistical scheme and show agreement with the replica symmetric 
approach. Furthermore, in an RSB1 treatment the critical storage capacity increases 
versus the RS values but up to 77 = 1 the increase is relatively small. 

The rest of this paper is organized as follows. In section 2 the SDS-model is shortly 
reviewed. In section 3 the RS approach to this model at zero temperature is outlined 
and a detailed comparison with the statistical method used in [|J is made. Section 
4 contains a discussion of the RSB1 solution. Some concluding remarks are given in 
section 5. 



2. The SDS-model 

Consider a network of N neurons which can take the values ±1 with equal 
probability. In this network we want to store p = aN patterns £f = ±1, % = 
l,2,...,N,fi = 1,2, ...,p that are supposed to be independent and identically 
distributed random variables with probability distribution Pr(£f) = |5(£f — l) + |5(£f + 

!)• 

Given a configuration a = (o"i, . . . , <tn), the local field hi of neuron i is 

h M = J ^ a i ' (*) 

where are the synaptic couplings given by 

^ = ^E^©(K) 2 -^ (2) 
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with the usual overlap order parameters defined by 

m " = JrJ2^ (3) 

i=l 

and rj > the threshold parameter. Due to the presence of the step function 0(-) only 
those terms where (m M ) 2 > rf /N contribute to the synaptic couplings. 

The neurons are updated asynchronously according to the well-known Glauber 
dynamics. We are interested in the zero temperature limit of this dynamics, which 
can be written as 

<T i (t+l) = sgD.[h i (tr(t))]. (4) 
For this deterministic dynamics an energy function has been found in ref. || 

H = -jZ((™n 2 -%)e((rn»r-?p). (5) 



3. A replica approach 

The energy function (|j) has been used in || to derive fixed-point equations for the 
relevant order parameters using the statistical mean- field scheme |6|, |7|]. The key idea of 
the latter calculation is to assume that the noise to which the small overlaps with the 
non-condensed patterns add up is Gaussian. 

In the following we apply the replica approach ^], || at zero temperature up to 
first-order breaking and compare our results with the statistical scheme. 



3.1. Replica symmetric results 

Following the standard approach we calculate the replica-symmetric free energy per 
neuron for the SDS-model at zero temperature as the limit (3 — > oo, with (5 the inverse 
temperature, of its temperature dependent form. For the latter we obtain as a function 
of the usual order parameters, i.e., the overlap, m 1 = m, with the condensed pattern 
fj, = 1, the Edwards-Anderson order parameter, q, and the residual overlap, r, with the 
non-condensed patterns // > 2, 

f( RS \m,r,q,f3) = f^ S \m,r, q,(3) + fV*> (q, (3) , (6) 



where 



fo RS \m,r,q,p) = ^- + -a/3r(l - q) + ^ 



ln(l-/3(l-g)) 



l-P(l-q) 



- -J J Dzln[2cosh/3(m+ y/ar z)} (7) 



1 



a. 



a 



Dz\yi 



l-Ierf(0 + (^))-ierf(0_(^)) 



'P. 



qz 



+ ^l-<?(l-»)expl|tf- 1 _ 



(erf(0 + (O)) + erf(^(O))) 



with 



[l-x(l-g)]?7± v /g^ 
V2(l-9)(l-x(l-g)) 



(9) 



and Dz = dz(2n)(~ 1 / 2 ') exp(— z 2 /2). In the above /o (•) is the free energy 
corresponding to the Hopfield model (77 = 0) while f^ RS \-) reflects the effect of the 
removal of the non-condensed patterns having a small correlation with the state of the 
system. Furthermore, the fixed-point equations are given by 



ar z) 



ar z) 



Q 



— + ——f {RS) (q 13) 



(10) 
(11) 
(12) 



In the limit 77 — > we find back the fixed-point equations for the Hopfield model, as 
we should. Furthermore, the change in the Hebb rule realized in eq. (0) manifests itself 
explicitly only in the order parameter r as one would expect. For zero temperature the 
fixed-point equations (10)-(12) reduce to 



m 



erf 



m 



V2 



ar 



1 — erf 



Jim /3(1 - q) 




r/exp 



(13) 
(14) 

(15) 



These results have to be compared with the statistical mean-field scheme of f|. But 
first we check the RS stability of this solution (13)-(14) by calculating, as an indication, 
the entropy of the replica symmetric phase. We find 



S — Sq + S v 



where 



So - ~ g 



ln(l - c) + 



(16) 



;i7) 
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Figure 1. The replica symmetric entropy S at zero temperature as a function of the 
threshold rj. 
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Again in the limit i] the expression ( |TE| ) reduces to the entropy of the Hopfield 
model (17) as given, e.g, in 0. As shown in Fig. 1, the entropy for the SDS-model 
is negative for all values of r\ indicating RS-breaking. We remark that for r] — > oo the 
entropy goes to — oo as — i]^ 1 exp(r/ 2 /2). To get an idea about the size of the breaking 
as a function of i] a first-order approximation (RSB1) will be performed in section 3.3. 



3.2. Comparison with the statistical scheme 

Comparing the fixed-point equations (13)-(14) with those of the statistical mean- 
field scheme derived in || (see egs. (23)- (26)) we find that they are different as soon as 
rj 7^ 0. This is illustrated by Fig. 2. It contrasts the situation for the Hopfield model 
where it is argued that the key assumption in the statistical approach mentioned 
above - the noise to which the small overlaps with the non-condensed patterns add 
up is Gaussian - oversimplifies and is most probably responsible for obtaining the 
results corresponding to the RS approximation. The reason that there is no such 
correspondence here is that the derivation in [||] not only invokes this key assumption but 
furthermore supposes that the overlap with the non-condensed patterns themselves, i.e., 
the (m^r), /i = 2, . . . ,p = aN where the brackets (•) indicate the thermal average, have 
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Figure 2. The critical capacity in the RS (full curve) and RSB1 (dotted curve) 
approximation as a function of the threshold r\. For comparison we also show the 
results of Ref . [5] (dashed-dotted curve). 



an identical normal distribution with mean zero and variance a 2 /N. (For convenience we 
write down explicitly the N-dependence in this subsection). These two assumptions are 
incompatible for rj ^ 0. Indeed, following closely the derivation in || |6|] by starting from 
the mean-field equations for the thermal average of the overlap with a non-condensed 
pattern, {m N ), and expanding it in a Taylor series to first order we arrive at 



1-/3(1-^)9 (m» N ) 2 - 



N 



(m» N ) = X N 



with 



X n = jjYl tanh/5(m7v + Vnj) 



N 



iV 



<1n = E tanh 2 /5(m 7V + Vnj) 



N 



(19) 

(20) 
(21) 
(22) 



Here rf N - is the noise part and we recall that (m N ) = m N is the overlap of the network 
with the condensed pattern 1. This expression shows that in the limit iV — > oo the 
relation between the distributions for the random variables {m v ) and rfc is no longer 
simply linear when rj ^ 0. In fact, starting from the key assumption that the noise 
has a Gaussian distribution, i.e., ~ J\f(0,ar) we find from (20) using the central 
limit theorem that linijv^oc \/NXn ~ A/"(0, q) with q the limit of (21), which is equal to 
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eq. (11). Furthermore, according to eq. (fOJ) we see that {m v N ) is a multi-valued function 
of the X^. Employing a standard geometrical Maxwell construction we obtain 



N(m» N ) 




NX N 
XX N 



> ^1-/3(1-^)77 
< y/l-/3(l-q N ) V 



(23) 



The expression ( P3|) clearly shows that the overlaps with the non-condensed patterns 
are not Gaussian distributed in the limit N — > oo. Using the correct distribution we 
precisely find eq. (14) for the order parameter r in the zero-temperature limit. Hence, we 
have shown complete equivalence between the statistical mean-field scheme using only 
the key assumption that the noise is Gaussian distributed, and the replica symmetric 
results. 



3.3. First-step breaking results 

From the observation on the entropy given in section 3.1 we expect RSB effects. 
In order to get an idea about the size of these effects with growing 77 we apply first- 
step RSB. We follow the standard approach (see, e.g., pT| ) by introducing the order 
parameters 

m M — m M Vat = 1, • • • , n 

<?a 7 = (! _ 9i)£a 7 + (?i - go)e« 7 + 9o Va =1 ... n ^ 



r, 



07 



(1 - ri)<5 a7 + (n - r )e a7 + r 



with n the number of replicas and {e Q7 },Va , ,7 = 1, • • • , n a (n X n) - matrix with 
elements 1 inside n/k diagonal blocks of size k and outside these blocks. 

The free energy per neuron can then be obtained after some tedious calculations 

/ (JBSrB1) (m,ft,«i,ro,ri,*,i9) 

= ti RSBl) (m, g , Ql , r , r 1; k, (3) + (q , Ql , k, (3) , (25) 

where the first term is given by 

f^ RSB1 \m,q ,q 1 ,r ,r 1 ,k,(3) = ^m 2 ^a(3{kq r + (1 - k)q 1 r 1 -rj 

+ ^Mi-^(i-?i)] 

- J Dzi In I^J Dz 2 cosh k (3(m + ^far^Zi + ^/a(ri - r ) z 2 ) | (26) 
and the second term reads 



exp 



(3z 2 



2(1- (3(1 - qi )) 



[1 - ierf (^+(^)) - ierf(^-C9))] 



7? 



(erf(^ + (0)) + erf(^_(0))) 



(27) 



with 



^±(z) 



[1 — x(l — (ft)]?? ± -2 



2(l- gi )(l-x(l- 9l )) 

In the limit 77 — > the expression reduces to the Hopfield RSB1 free energy as calculated, 
e.g, in 



12, tt a 14 



Again in the following we are only interested in the zero temperature results. From 
the limit (3 — > oo of expression (27) we can obtain the fixed-point equations for the 
relevant order parameters. Since the way to derive these formula is standard and since 
their explicit expressions are algebraically complicated we do not write them down. 

The zero temperature critical capacity a^ RSB1 ' as a function of 77 is presented in 
Fig. 2. For 77 = we confirm the result a^ RSB1 ' = 0.13819 found in [13[ [14|]. For growing 
77 the results for a^ RSB1 ^ and a^ RS ^ start deviating more. In view of the results on the 
entropy (see Fig. 1) we expect that the difference keeps growing. Since the calculations 
are very tedious and since in the literature one is mostly interested in values of 77 smaller 
than 1 |5], |, [L0| we have plotted results up to 77 = 1. For 77 = 1, e.g., we find 0.16658 



for the RSB1 critical capacity versus 0.16384 for the RS critical capacity. The results 
of [H overestimate this value. 



4. Concluding remarks 

The replica method is applied to an existing neural network model with state- 
dependent couplings. Only those patterns having a correlation with the state of the 
system greater than a threshold 77 contribute to the couplings. 

The free energy is obtained and the fixed-point equations are studied at zero 
temperature. It is shown that the fixed-point equations in the replica symmetric 
approximation coincide with these found by the so-called heuristically motivated 
statistical mean- field scheme developed in || |7]], provided one does not make the 
additional assumption that the overlap with the non-condensed patterns is Gaussian. 
This assumption, made in the literature, is totally unnecessary and is even incompatible 
with the key assumption of the statistical method that the noise induced by the non- 
condensed patterns is Gaussian. 

The critical storage capacity at zero temperature is calculated as a function of 
the threshold 77 and compared with the values obtained in the literature on the basis 
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of the statistical method with the extra Gaussian assumption for the overlaps. Since a 
calculation of the entropy indicates that replica symmetry is broken at zero temperature 
for all values of rj a first order replica symmetry breaking calculation has been performed. 
The critical storage capacity increases versus the replica symmetric values but up to 
77 = 1 the increase is relatively small. 
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